Repository navigation
Preserve extra Arrow field definitions during weak schema validation - #298
Closed
pengpengyi92 wants to merge 1 commit into
Closed
pengpengyi92 wants to merge 1 commit into
pengpengyi92 wants to merge 1 commit into
Conversation
Signed-off-by: Pengyi Peng <pengpengyi92@users.noreply.github.com>
pengpengyi92
requested review from
feussy,
hintse,
ptomecek and
timkpaine
as code owners
November 14, 2026 10:28
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #298 +/- ##
=======================================
Coverage 93.85% 93.85%
=======================================
Files 205 205
Lines 22944 22974 +30
Branches 1511 1512 +1
=======================================
+ Hits 21533 21563 +30
Misses 1120 1120
Partials 291 291 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
4 of 11 tasks
Member
|
Thanks for the PR, i've consolidated several open ones in #299 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
When validating an
ArrowTable[ArrowSchema.make(schema, strict=False)], extra columns are kept, but their field definitions are not: appending them by name drops field metadata and resetsnullable=FalsetoTrue.For example, an extra
pa.field("d", pa.int64(), nullable=False, metadata={b"unit": b"shares"})becomes a nullable field with no metadata after validation. The values remain present, so value-only assertions do not detect the loss.Pass the original
pa.Fieldtoappend_columninstead. Required columns still cast to the declared schema; extra column order and values, filter mode, and strict rejection remain unchanged. No dependency, public API, or documentation-contract change.Regression coverage includes multiple metadata-bearing extras, a nullable column containing null, non-nullability, empty record batches, and multi-batch input with required-column casting. Schema checks explicitly include metadata. This is separate from #295, which preserves table-level metadata in the annotation utility.
Type of Change
Validation
On base
83f2b0ac7128c08c92f2b6fd60f58723ac47635a, the new cases fail on metadata/non-nullability loss, including both empty and chunked inputs. With this one-line fix:ccflow(208 files).make test, documentation lint and performance measurements were not run locally.Checklist
make lint; Python portion passed, docs lint not run)make test; scoped native tests passed as above)AI assistance: Codex assisted with investigation, the original patch, regression tests, and self-review. No separate human or maintainer approval is claimed. The contributor authorized DCO sign-off for this original contribution after verification; no private code or data is included.
Thanks for reviewing; happy to adjust this to the preferred schema-preservation contract.